Andrew Cater: Afternoon talks - MiniDebConf ARM Cambridge - Day 1
Thanks to all involved with Arm, Codethink and Pexip for hosting and sponsorship without which this would not have been possible.
Photo by Pixabay |
Given a typical install of 3 generic kernel ABIs in the default configuration on a regular-sized VM (2 CPU cores 8GB of RAM) the following metrics are achieved in Ubuntu 23.10 versus Ubuntu 22.04 LTS:
2x less disk space used (1,417MB vs 2,940MB, including initrd)
3x less peak RAM usage for the initrd boot (68MB vs 204MB)
0.5x increase in download size (949MB vs 600MB)
2.5x faster initrd generation (4.5s vs 11.3s)
approximately the same total time (103s vs 98s, hardware dependent)
For minimal cloud images that do not install either linux-firmware or modules extra the numbers are:
1.3x less disk space used (548MB vs 742MB)
2.2x less peak RAM usage for initrd boot (27MB vs 62MB)
0.4x increase in download size (207MB vs 146MB)
Hopefully, the compromise of download size, relative to the disk space & initrd savings is a win for the majority of platforms and use cases. For users on extremely expensive and metered connections, the likely best saving is to receive air-gapped updates or skip updates.
This was achieved by precompressing kernel modules & firmware files with the maximum level of Zstd compression at package build time; making actual .deb files uncompressed; assembling the initrd using split cpio archives - uncompressed for the pre-compressed files, whilst compressing only the userspace portions of the initrd; enabling in-kernel module decompression support with matching kmod; fixing bugs in all of the above, and landing all of these things in time for the feature freeze. Whilst leveraging the experience and some of the design choices implementations we have already been shipping on Ubuntu Core. Some of these changes are backported to Jammy, but only enough to support smooth upgrades to Mantic and later. Complete gains are only possible to experience on Mantic and later.
The discovered bugs in kernel module loading code likely affect systems that use LoadPin LSM with kernel space module uncompression as used on ChromeOS systems. Hopefully, Kees Cook or other ChromeOS developers pick up the kernel fixes from the stable trees. Or you know, just use Ubuntu kernels as they do get fixes and features like these first.
The team that designed and delivered these changes is large: Benjamin Drung, Andrea Righi, Juerg Haefliger, Julian Andres Klode, Steve Langasek, Michael Hudson-Doyle, Robert Kratky, Adrien Nader, Tim Gardner, Roxana Nicolescu - and myself Dimitri John Ledkov ensuring the most optimal solution is implemented, everything lands on time, and even implementing portions of the final solution.
Hi, It's me, I am a Staff Engineer at Canonical and we are hiring https://canonical.com/careers.
Lots of additional technical details and benchmarks on a huge range of diverse hardware and architectures, and bikeshedding all the things below:
Welcome to the September 2023 report from the Reproducible Builds project
In these reports, we outline the most important things that we have been up to over the past month. As a quick recap, whilst anyone may inspect the source code of free software for malicious flaws, almost all software is distributed to end users as pre-compiled binaries.
Andreas Herrmann gave a talk at All Systems Go 2023 titled Fast, correct, reproducible builds with Nix and Bazel . Quoting from the talk description:
You will be introduced to Google s open source build system Bazel, and will learn how it provides fast builds, how correctness and reproducibility is relevant, and how Bazel tries to ensure correctness. But, we will also see where Bazel falls short in ensuring correctness and reproducibility. You will [also] learn about the purely functional package manager Nix and how it approaches correctness and build isolation. And we will see where Bazel has an advantage over Nix when it comes to providing fast feedback during development.Andreas also shows how you can get the best of both worlds and combine Nix and Bazel, too. A video of the talk is available.
file(1)
version 5.45 [ ] and updated some documentation [ ]. In addition, Vagrant Cascadian extended support for GNU Guix [ ][ ] and updated the version in that distribution as well. [ ].
BUILDSPEC.md
file. [ ] And Fay Stegerman fixed the builds failing because of a YAML syntax error.
.dsc
file modulo the GPG signature . This month, however, Russ Allbery closed the bug due to concerns about the viability of source reproducibility.
linuxsampler
(benchmarking issue)antlr3
(date)rpm
(embeds too many build details)seamonkey
(date)conky
(date and ordering-related issue)lsp-plugins-shared
(date/copyright year issue)build-compare
helix
(ASLR-related non-determinism)intel-graphics-compiler
(ASLR)sphinxcontrib-mermaid
.mkdocs-material
.apophenia
.lapackpp
.blaspp
.mysql-connector-java
, java-21-openjdk
, apache-ivy
, maven-assembly-plugin
, eclipse
, antlr3
, groovy18
, hbci4java
, ini4j
, hppc
, checkstyle
, glassfish-jaxb
, tycho
, xmvn
, mockito
, languagetool
, json-lib
, jnr-unixsocket
, jnr-ffi
, jnr-enxio
, jboss-jaxrs-2.0-api
, istack-commons
, rxtx-java
, glassfish-jaxb
, glassfish-hk2
, findbugs
, docker-client-java
, maven
, xmvn-connector-ivy
, xmlstreambuffer
, checkstyle
, cglib
, bean-validation-api
, aws-sdk-java
, javapackages-tools
, ant
, scala
, osgi-service-log
, jmdns
, xml-security
, super-csv
, osgi-service-jdbc
, msv
, junit5
, jsr-311
, jersey
, itextpdf
, httpcomponents-asyncclient
, ed25519-java
, jnacl
, javaparser
, picocli
, freemarker
, extra166y
, javaparser
, xstream
, woodstox-core
, uom-lib
, unit-api
, uncommons-maths
, tycho
, treelayout
, tiger-types
, super-csv
, stax-ex
, stax2-api
, sqlite-jdbc
, reflectasm
, prometheus-simpleclient-java
, powermock
, paranamer
, opennlp
, netty3
, mybatis
, morfologik-stemming
, minlog
, maven-archetype
, mariadb-java-client
, logback
, kryo
, jsonp
, jopt-simple
, jnr-posix
, jnr-constants
, jnr-a64asm
, jfreechart
, jffi
, jetty-schemas
, jetty-minimal
, jeromq
, jctools
, jcsp
, jboss-websocket-1.0-api
, jboss-marshalling
, jboss-logmanager
, jboss-logging
, javaewah
, jatl
, janino
, jackson-modules-base
, jackson-jaxrs-providers
, jackson-datatypes-collections
, jackson-dataformat-xml
, jackson-dataformats-text
, jackson-dataformats-binary
, indriya
, google-gson
, glassfish-websocket-api
, glassfish-transaction-api
, glassfish-jsp
, glassfish-jax-rs-api
, glassfish-hk2
, glassfish-fastinfoset
, felix-scr
, felix-gogo-shell
, felix-gogo-command
, disruptor
, apache-commons-ognl
, apache-commons-math
, apache-commons-csv
, antlr4
, jettison
, sisu
, maven
armhf
and i386
builds due to Debian bug #1052257. [ ][ ][ ][ ]ionice
priority. [ ]dinstall
again. [ ]schroot
running the tested suite. [ ][ ]diffoscope --version
(as suggested by Fay Stegerman on our mailing list) [ ], worked on an openQA credential issue [ ] and also made some changes to the machine-readable reproducible metadata, reproducible-tracker.json
[ ]. Lastly, Roland Clobus added instructions for manual configuration of the openQA secrets [ ].
#reproducible-builds
on irc.oftc.net
.
rb-general@lists.reproducible-builds.org
ubuntu-release-upgrader
changes your sources.list, and applies various
quirks to the upgrade.
In this post, I want to look not at the quirk aspects but discuss how dependency
solving should differ between intra-release and inter-release upgrades.
Previous solver projects (such as Mancoosi) operated under the assumption that minimizing
the number of changes performed should ultimately be the main goal of a solver. This makes
sense as every change causes risks.
However it ignores a different risk, which especially applies when upgrading from one
distribution release to a newer one: Increasing divergence from the norm.
Consider a person installs foo
in Debian 12. foo
depends on a b
, so a
will
be automatically installed to satisfy the dependency. A release later, a
has some
known issues and b
is prefered, the dependency now reads: b a
.
A classic solver would continue to keep a
installed because it was installed before,
leading upgraded installs to have foo, a
installed whereas new systems have foo, b
installed. As systems get upgraded over and over, they continue to diverge further and
further from new installs to the point that it adds substantial support effort.
My proposal for the new APT solver is that when we perform release upgrades, we forget
which packages where previously automatically installed.
We effectively perform a normalization: All systems with the same set of manually installed packages will end up with the same set of automatically installed packages.
Consider the solving starting with an empty set and then installing the latest version of each previously manually installed package: It will see now that foo
depends
b a
and install b
(and a
will be removed later on as its not part of the solution).
Another case of divergence is Suggests
handling. Consider that foo
also Suggests
s
. You now install another package bar
that depends s
, hence s
gets installed.
Upon removing bar
, s
is not being removed automatically because foo
still suggests
it (and you may have grown used to foo
s integration of s
). This is because apt considers
Suggests to be important - they won t be automatically installed, but will not be automatically
removed.
In Ubuntu, we unset that policy on release upgrades to normalize the systems. The reasoning
for that is simple: While you may have grown to use s
as part of foo
during the release,
an upgrade to the next release already is big enough that removing s
is going to have
less of an impact - breakage of workflows is expected between release upgrades.
I believe that apt release-upgrade
will benefit from both of these design choices,
and in the end it boils down to a simple mantra:
# Load configuration files for the default server block.
include /etc/nginx/default.d/*.conf;
location /
autoindex on;
autoindex_exact_size off;
autoindex_format html;
autoindex_localtime off;
# First attempt to serve request as file, then
# as directory, then fall back to displaying a 404.
try_files $uri $uri/ =404;
[Unit]
Description=Rocky Linux Mirroring script
[Service]
Type=simple
User=mirror
Group=mirror
ExecStart=/usr/local/bin/rockylinux
[Install]
WantedBy=multi-user.target
[Unit]
Description=Run Rocky Linux mirroring script daily
[Timer]
OnCalendar=*-*-* 08:13:00
OnCalendar=*-*-* 22:13:00
Persistent=true
[Install]
WantedBy=timers.target
This is a very long script in total.#!/bin/env bash
#
# mirrorsync - Synchronize a Rocky Linux mirror
# By: Dennis Koerner <koerner@netzwerge.de>
#
# The latest version of this script can be found at:
# https://github.com/rocky-linux/rocky-tools
#
# Please read https://docs.rockylinux.org/en/rocky/8/guides/add_mirror_manager
# for further information on setting up a Rocky mirror.
#
# Copyright (c) 2021 Rocky Enterprise Software Foundation
Logfile looks something like this: the single time spec file is used to check whether another rsync needs to be run# A complete list of mirrors can be found at
# https://mirrors.rockylinux.org/mirrormanager/mirrors/Rocky
src="mirrors.vinters.com::rocky"
# Your local path. Change to whatever fits your system.
# $mirrormodule is also used in syslog output.
mirrormodule="rocky-linux"
dst="/srv/$ mirrormodule "
filelistfile="fullfiletimelist-rocky"
lockfile="/home/mirror/rocky.lockfile"
logfile="/home/mirror/rocky.log"
deleting 9.1/plus/x86_64/os/repodata/3585b8b5-90e0-4856-9df2-95f646bc62c7-PRIMARY.xml.gzIt was essentially easier to store fullfiletimelist-rocky in /home/mirror than anywhere else.
sent 606,565 bytes received 38,808,194,155 bytes 44,839,746.64 bytes/sec
total size is 1,072,593,052,385 speedup is 27.64
End: Fri 27 Jan 2023 08:27:49 GMT
fullfiletimelist-rocky unchanged. Not updating at Fri 27 Jan 2023 22:13:16 GMT
fullfiletimelist-rocky unchanged. Not updating at Sat 28 Jan 2023 08:13:16 GMT
Next.